Worker heartbeat: New in-memory metrics mechism, plumb rest of heartbeat data #1023

yuandrew · 2025-09-29T22:14:44Z

What was changed

NOTE: targeting worker-heartbeat feature branch.

New in memory mechanism to keep track of certain metrics for worker heartbeating.

Why?

Checklist

Closes
How was this tested:

Any docs updates needed?

Note

Introduce in-memory metrics for worker heartbeats, populate/export detailed heartbeat, poller, and slot info, track slot-supplier kind, and add DescribeWorker/SetWorkerDeploymentManager APIs with tests.

Telemetry/Metrics:
- Add in-memory metric tracking (HeartbeatMetricType, WorkerHeartbeatMetrics) and wire into Counter/Gauge/HistogramDuration via *_with_in_memory helpers.
- Extend MetricsContext::Instruments to record heartbeat-related counters/gauges/histograms; add NoOp attributes support.
Worker/Runtime:
- Populate WorkerHeartbeat with SDK identity, times, poller info (incl. last successful poll), slot usage, sticky cache, plugins, and status.
- Track last successful poll times in poll buffers/scaler; add worker status and expose worker_instance_key() on API.
- Add WorkerConfig fields: plugins, skip_client_worker_set_check.
- Track slot-supplier kind (Fixed/ResourceBased/custom) and include in heartbeat.
- Improve resource-based tuner sysinfo with background refresh thread.
Client/Registry:
- ClientWorkerSet::register accepts skip_client_worker_set_check; heartbeat callback uses Arc.
- Client sets heartbeat client-derived fields and computes last-interval slot counters.
APIs/Protos/OpenAPI:
- Add DescribeWorker RPC and wire through client/C-bridge.
- Add SetWorkerDeploymentManager RPC and related messages.
- Update multiple proto/OpenAPI structs (namespace capabilities, deployment info, history-reverse doc, priority doc, etc.).
Tests:
- New integration tests for worker heartbeat data/metrics, tuner behavior, multiple workers, and config flags; adjust existing tests for API changes.

^{Written by Cursor Bugbot for commit fc7f839. This will update automatically on new commits. Configure here.}

core-api/src/telemetry/metrics.rs

core-c-bridge/src/metric.rs

core/src/pollers/poll_buffer.rs

core/src/telemetry/metrics.rs

core/src/worker/client.rs

core/src/worker/mod.rs

…g, counter_with_in_mem and and_then()

…, remove stale metric previously added

…ing integ test

client/src/lib.rs

core-api/src/telemetry/metrics.rs

Sushisource · 2025-10-07T22:58:56Z

core-api/src/telemetry/metrics.rs

+impl WorkerHeartbeatMetrics {
+    pub fn get_metric(&self, name: &str) -> Option<HeartbeatMetricType> {
+        match name {
+            "sticky_cache_size" => Some(HeartbeatMetricType::Individual(


All these names are duplicative of some existing metric name in core/src/telemetry/metrics.rs, which is calling through into this anyway. Rather than having all the fields be pub, I think it'd be safer to make getters for each, individually, that do the same thing this match does.

that do the same thing this match does.

Not sure I follow this part, happy to make getters so all fields aren't pub, but it sounds like you're suggesting a change to the match itself? We need some way to map the str name to the struct field. Are you suggesting moving this whole fn over to MetricParameters?

core/src/worker/tuner/resource_based.rs

core/src/worker/mod.rs

tests/integ_tests/worker_heartbeat_tests.rs

…ers directly, WithLabel API improvement

core/src/worker/mod.rs

…ing, more tests

client/src/worker_registry/mod.rs

Sushisource

Looks like some of the tests are failing too

Sushisource · 2025-10-14T21:08:20Z

core-api/src/worker.rs

    #[builder(default)]
    pub plugins: Vec<PluginInfo>,
+
+    /// Skips the single worker+client+namespace+task_queue check


GH not letting me suggest this line for whatever crazy reason, but:

"Skips the check that enforces only one worker per client may exist with the same namespace/task queue combination"

But, actually I guess it's literally disabling registration entirely. If this is true, even if there are no other conflicting workers, will it end up disabling the worker heartbeating? Definitely shouldn't if it does.

Yikes! That's def not what we want. Added a test to verify this behavior

core/src/worker/mod.rs

core/src/worker/client.rs

yuandrew added 2 commits September 29, 2025 15:13

plumb in memory metrics

c32d6c9

simplify worker::new(), fix some heartbeat metrics, new test file

0ad6be2

Sushisource reviewed Sep 30, 2025

View reviewed changes

Sushisource mentioned this pull request Sep 30, 2025

[Feature Request] Support CPU Cgroup Limits in Resource-based tuner / Confirm memory functionality #903

Open

yuandrew added 5 commits October 1, 2025 11:31

CounterImpl, final_heartbeat, more specific metric label dbg_panic ms…

d314a35

…g, counter_with_in_mem and and_then()

Support in-mem metrics when metrics aren't configured

b00b80e

Move sys_info refresh to dedicated thread, use tuner's existing sys info

894d916

Format, AtomicCell

ef96616

Fix unit test

0350f91

yuandrew marked this pull request as ready for review October 5, 2025 03:34

yuandrew requested a review from a team as a code owner October 5, 2025 03:34

This comment was marked as outdated.

Sign in to view

Set dynamic config for WorkerHeartbeatsEnabled and ListWorkersEnabled…

fe3956c

…, remove stale metric previously added

This comment was marked as outdated.

Sign in to view

Should not expect heartbeat nexus worker in metrics for non-heartbeat…

d956784

…ing integ test

This comment was marked as outdated.

Sign in to view

Sushisource reviewed Oct 7, 2025

View reviewed changes

recv_timeout instead of thread::sleep, use WorkflowService::list_work…

98e778d

…ers directly, WithLabel API improvement

Sushisource reviewed Oct 8, 2025

View reviewed changes

core/src/worker/mod.rs Outdated Show resolved Hide resolved

MetricAttributes::NoOp, add mechanism to ignore dupe workers for test…

96e0186

…ing, more tests

Sushisource reviewed Oct 10, 2025

View reviewed changes

client/src/worker_registry/mod.rs Outdated Show resolved Hide resolved

More tests, sticky cache miss, plugins

e1ad0ae

Sushisource reviewed Oct 14, 2025

View reviewed changes

Formatting, fix skip_client_worker_set_check

8f42387

cursor bot reviewed Oct 14, 2025

View reviewed changes

core/src/worker/mod.rs Show resolved Hide resolved

yuandrew added 2 commits October 14, 2025 15:44

Cursor found a bug

d54a3eb

Lower sleep time, add print for debugging

13d981c

cursor bot reviewed Oct 14, 2025

View reviewed changes

core/src/worker/mod.rs Show resolved Hide resolved

yuandrew force-pushed the worker-heartbeat-telemetry branch from 398b5ce to 33fd78d Compare October 14, 2025 23:24

cursor bot reviewed Oct 14, 2025

View reviewed changes

core/src/worker/client.rs Outdated Show resolved Hide resolved

more prints

f1a3634

yuandrew force-pushed the worker-heartbeat-telemetry branch from 33fd78d to f1a3634 Compare October 14, 2025 23:53

cursor bot reviewed Oct 14, 2025

View reviewed changes

core/src/worker/client.rs Outdated Show resolved Hide resolved

use semaphores for worker_heartbeat_failure_metrics

6d3a909

cursor bot reviewed Oct 15, 2025

View reviewed changes

core/src/worker/client.rs Outdated Show resolved Hide resolved

skip_client_worker_set_check for all integ workers

a29d2df

cursor bot reviewed Oct 15, 2025

View reviewed changes

core/src/worker/client.rs Outdated Show resolved Hide resolved

Can't use tokio semaphore in workflow code

fc7f839

Worker heartbeat: New in-memory metrics mechism, plumb rest of heartbeat data #1023

Are you sure you want to change the base?

Worker heartbeat: New in-memory metrics mechism, plumb rest of heartbeat data #1023

Uh oh!

Conversation

yuandrew commented Sep 29, 2025 • edited by cursor bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What was changed

Why?

Checklist

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Sushisource Oct 7, 2025

Choose a reason for hiding this comment

Uh oh!

yuandrew Oct 8, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Sushisource left a comment

Choose a reason for hiding this comment

Uh oh!

Sushisource Oct 14, 2025

Choose a reason for hiding this comment

Uh oh!

Sushisource Oct 14, 2025

Choose a reason for hiding this comment

Uh oh!

yuandrew Oct 14, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

yuandrew commented Sep 29, 2025 •

edited by cursor bot

Loading